Continuous electronic zoom for an imaging system with multiple imaging devices having different fixed fov

ABSTRACT

A method for continuous electronic zoom in a computerized image acquisition system, the system having a wide image acquisition device and a tele image acquisition device having a tele image sensor array coupled with a tele lens having a narrow FOV, and a tele electronic zoom. The method includes providing a user of the image acquisition device with a zoom selecting control, thereby obtaining a requested zoom, selecting one of the image acquisition devices based on the requested zoom and acquiring an image frame, thereby obtaining an acquired image frame, and performing digitally zoom on the acquired image frame, thereby obtaining an acquired image frame with the requested zoom. The alignment between the wide image sensor array and the tele image sensor array is computed, to facilitate continuous electronic zoom with uninterrupted imaging, when switching back and forth between the wide image sensor array and the tele image sensor array.

RELATED APPLICATION

The present application claims the benefit of U.S. provisionalapplication 61/167,226 filed on Apr. 7, 2009, the disclosure of which isincorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates to an electronic zoom for imaging systems,and more particularly, the present invention relates to a continuouselectronic zoom for an image acquisition system, the system includingmultiple imaging devices having different fixed FOV.

BACKGROUND OF THE INVENTION AND PRIOR ART

Digital zoom is a method of narrowing the apparent angle of view of adigital still or video image. Electronic zoom is accomplished bycropping an image down to a centered area of the image with the sameaspect ratio as the original, and usually also interpolating the resultback up to the pixel dimensions of the original. It is accomplishedelectronically, without any adjustment of the camera's optics, and nooptical resolution is gained in the process. Typically some informationis lost in the process.

In video streams (such as PAL, NTSC, SECAM, 656, etc.) the imageresolution is known, and by using image sensors having substantiallyhigher resolution, one can perform lossless electronic zoom. The ratiobetween the image sensor resolution and the output resolution dictatesthe lossless electronic zoom range. For example, having a 5 Megapixel,2592×1944, image sensor array and an output resolution frame of 400×300yields maximal lossless electronic zoom of 6.48:

-   -   2592/400=6.48,    -   1944/300=6.48.

Typically, a camera with a large dynamic zoom range requires heavy andexpensive lenses, as well as complex design. Electronic zoom does notneed moving mechanical elements, as does optical zoom.

There is a need for and it would be advantageous to have image sensors,having static, light weight electronic zoom and a large lossless zoomingrange.

SUMMARY OF THE INVENTION

The present invention describes a continuous electronic zoom for animage acquisition system, having multiple imaging devices each with adifferent fixed field of view (FOV). Using two (or more) image sensors,having different fixed FOV, facilitates a light weight electronic zoomwith a large lossless zooming range. For example, a first image sensorhas a 60° angle of view and a second image sensor has a 60° angle ofview. Therefore, Wide_FOV=Narrow_FOV*6. Hence, switching between theimage sensors provide a lossless electronic zoom of 6²=36. This losslesselectronic zoom is also referred to herein, as the optimal zoom:

Optimal_Zoom=(Wide_FOV/Narrow_FOV)².

It should be noted that to obtain similar zoom (×36) by optical means,for an output resolution frame of 400×300, the needed image sensor arrayis:

36*400=14400,

36*300=10800.

14400*10800=155,520,000.

Hence, to obtain a zoom of ×36 by optical means, for an outputresolution frame of 400×300, one needs a 155 Megapixel, 14400×10800,image sensor array.

According to teachings of the present invention, there is provided amethod for continuous electronic zoom in a computerized imageacquisition system, the system having multiple optical image acquisitiondevices each with a FOV. The method includes providing a first imageacquisition device having a first image sensor array coupled with afirst lens having a first FOV, typically a wide FOV , and a firstelectronic zoom. The method further includes providing a second imageacquisition device having a second image sensor array coupled with asecond lens having a second FOV, typically a narrow FOV, and a secondelectronic zoom. Typically, the angle of view of the first FOV is widerthan the angle of view of the second FOV. At least a portion of theenvironment, viewed from within the second FOV of the second imageacquisition device, overlaps the environment viewed from within thefirst FOV of the first image acquisition device. The method furtherincludes computing the alignment between the first image sensor arrayand the second image sensor array, whereby determining an X-coordinateoffset, a Y-coordinate offset and optionally, a Z-rotation offset of thecorrelation between the first image sensor array and the second imagesensor array.

The method further includes the steps of providing a user of the imageacquisition device with a zoom selecting control, thereby obtaining arequested zoom, selecting one of the image acquisition devices based onthe requested zoom, acquiring an image frame with the selected imageacquisition device, thereby obtaining an acquired image frame, andperforming digitally zoom on the acquired image frame, thereby obtainingan acquired image frame with the requested zoom. The calibration of thealignment, between the first image sensor array and the second imagesensor array, facilitates continuous electronic zoom with uninterruptedimaging, when switching back and forth between the first image sensorarray and the second image sensor array. Preferably the electroniccalibration is performed with sub-pixel accuracy.

Optionally, the computerized image acquisition system is configured toprovide zooming functions selected from the group consisting of a binfunction and a skip function. The selecting of the image acquisitiondevice includes selecting the parameters of the bin and/or skipfunctions, wherein the method further includes the step of applying theselected bin/skip functions to the acquired image frame, before theperforming of the digital zoom step.

In variations of the present invention, the image sensor arrays arefocused to the infinite.

Optionally, the first lens is a focus adjustable lens.

Optionally, the second lens is a focus adjustable lens.

Optionally, the second lens is a zoom lens.

In image acquisition systems having more than two imaging devices, theelectronic calibration step is performed on each pair of adjacentlydisposed image sensor arrays.

In variations of the present invention, the first image acquisitiondevice and the second image acquisition device are coupled with a mutualfront lens and a beam splitter, wherein one portion of the lightreaching the beam splitter is directed towards the first image sensorarray and the remainder portion of the light reaching the beam splitteris directed towards the second image sensor array.

In embodiments of the present invention, the first image sensor array isa color sensor and the second image sensor array is a monochrome sensor,wherein a colored image frame is acquired by the first image sensorarray, a monochrome image frame is acquired by the second image sensorarray, wherein the colored image frame and the monochrome image frameare fused to form a high resolution colored image frame. In preferredembodiments of the present invention, the angle of view of the first FOVis wider than the angle of view of the second FOV. However, in variationof the present invention, the angle of view of the first FOV issubstantially equal to the angle of view of the second FOV.

Optionally, the fusion of the colored image frame and the monochromeimage frame includes the step of computing color values for the highresolution pixels of the monochrome image frame from the respective lowresolution pixels of the colored image frame. Optionally, the computingof color values is performed in sub pixel accuracy.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will become fully understood from the detaileddescription given herein below and the accompanying drawings, which aregiven by way of illustration and example only and thus not limitative ofthe present invention, and wherein:

FIG. 1 is a block diagram illustration of another zoom controlsub-system for an image acquisition system, according to variations ofthe present invention;

FIG. 2 is a schematic flow diagram chart that outlines the successivesteps of the continuous zoom process, according to embodiments of thepresent invention;

FIG. 3 is a block diagram illustration of a zoom control sub-system foran image acquisition system, according to variations of the presentinvention;

FIG. 4 is a schematic flow diagram chart that outlines the successivesteps of the continuous zoom process, according to variations of thepresent invention, include using bin/skip functions;

FIGS. 5 a and 5 b illustrate examples of beam splitter configurationsfor image acquisition systems, according to embodiments of the presentinvention;

FIG. 6 is a block diagram illustration of a camera system, according toembodiments of the present invention, including a color image sensorhaving wide FOV and a color image sensor having narrow FOV; and

FIG. 7 is a block diagram illustration of another zoom controlsub-system for a color image acquisition system, according to variationsof the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Before explaining embodiments of the invention in detail, it is to beunderstood that the invention is not limited in its application to thedetails of construction and the arrangement of the components set forthin the host description or illustrated in the drawings.

Unless otherwise defined, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art of the invention belongs. The methods and examples providedherein are illustrative only and not intended to be limiting.

It should be noted that in general, the present invention is described,with no limitations, in terms of an image acquisition system having twoimage acquisition devices. But the present invention is not limited totwo image acquisition devices, and in variations of the presentinvention, the image acquisition system can be similarly embodied withthree image acquisition devices and more.

Reference is made to FIG. 1, which is a block diagram illustration of azoom control sub-system 100 for an image acquisition system, accordingto preferred embodiments of the present invention. Zoom controlsub-system 100 includes multiple image sensors, each with a fixed andpreferably different FOV, configured to provide continuous electroniczoom capabilities with uninterrupted, when switching back and forthbetween the image sensors.

Zoom control sub-system 100 includes a tele image sensor 110 coupledwith a narrow lens 120 having a predesigned FOV 140, a wide image sensor112 coupled with a wide lens 122 having a predesigned FOV 142, a zoomcontrol module 130 and an image sensor selector 150. An object 20 isviewed from both tele image sensor 110 and wide image sensor 112,whereas the object is magnified in tele image sensor 110 with respect towide image sensor 112, by a predesigned factor. In the optimalconfiguration, the FOV of wide image sensor 112 can be calculated bymultiplying the FOV of tele image sensor 110 by the optimal zoom ofimage sensors 110 and 112. Tele image sensor 110 and wide image sensor112 are adjacently disposed, such that at least a portion of theenvironment viewed from within the narrow FOV of tele image acquisitiondevice 110 overlaps the environment viewed from within the wide FOV ofwide image acquisition device 112.

Before using zoom control sub-system 100, an electronically calibratingis performed to determine the alignment offsets between wide imagesensor array 110 and tele image sensor array 112. Typically, since thespatial offsets between wide image sensor array 110 and tele imagesensor array 112 are fixed, the electronic calibration step is performedone time, after the manufacturing of the image acquisition system andbefore the first use. The electronic calibration yields an X-coordinateoffset, a Y-coordinate offset and optionally, a Z-coordinate rotationaloffset of the correlation between wide image sensor array 110 and teleimage sensor array 112. Preferably, all three aforementioned offsetvalues are computed in sub-pixel accuracy. It should be noted that forimage acquisition systems with more than two image sensors, theelectronic calibration step is performed on each pair of adjacentlydisposed image sensor arrays.

Zoom control circuit 130 receives a required zoom from an operator ofthe image acquisition system, and selects the relevant image sensor (110and 112) by activating image sensor selector 150 position. The relevantcamera zoom factor is calculated by zoom control unit 130.

An aspect of the present invention is to provide methods facilitatingcontinuous electronic zoom capabilities with uninterrupted imaging,performed by an image acquisition system having multiple image sensors,each with a fixed and preferably different FOV. The continuouselectronic zoom with uninterrupted imaging is also maintained whenswitching back and forth between adjacently disposed image sensors.

Reference is also made to FIG. 2, which is a schematic flow diagramchart that outlines the successive steps of an example continuous zoomprocess 200, according to embodiments of the present invention,performed on image acquisition system, having a zoom control sub-systemsuch as zoom control sub-system 100. Process 200 includes the flowingsteps:

Step 210: providing a wide image acquisition device and a tele imageacquisition device.

-   -   Multiple optical image acquisition devices can be used, but for        description clarity, with no limitation, the method will be        described in terms of two image acquisition devices: wide image        acquisition device and a tele image acquisition device.    -   Both image acquisition devices (110 and 112) include an image        sensor array coupled with a lens (120 and 122, respectively),        providing a fixed FOV (tele FOV 140 and wide FOV 142,        respectively). Preferably, wide FOV 142 is substantially wider        than narrow FOV 140.    -   The image acquisition devices are adjacently disposed, such that        at least a portion of the environment, viewed from within narrow        FOV 140 of the tele image acquisition device 110, overlaps the        environment viewed from within the wide FOV 142 of wide image        acquisition device 112.        Step 220: determining alignment offsets.    -   Before using zoom control sub-system 100, an electronically        calibrating is performed to determine the alignment offsets        between wide image sensor array 110 and tele image sensor array        112. Typically, since the spatial offsets between wide image        sensor array 110 and tele image sensor array 112 are fixed, the        electronic calibration step is performed one time, after the        manufacturing of the image acquisition system and before the        first use. The electronic calibration yields an X-coordinate        offset and a Y-coordinate offset of the correlation between wide        image sensor array 110 and tele image sensor array 112.        Preferably, the X-coordinate offset and the Y-coordinate offset        are computed in sub-pixel accuracy. It should be noted that for        image acquisition systems with more than two image sensors, the        electronic calibration step is performed on each pair of        adjacently disposed image sensor arrays.        Step 230: zoom selection.    -   A user of the image acquisition selects the required zoom.        Step 240: selecting an image acquisition device.    -   The zoom control 130 selects an image acquisition device with        the having a zoom more proximal to the requested zoom.        Step 250: acquiring an image frame.    -   An image frame is acquired by the selected image acquisition        device.        Step 260: resampling the acquired image frame to the requested        zoom.    -   The zoom control 130 computes the zoom factor between the fixed        zoom of the selected image acquisition device and the requested        zoom. Based on the computed factor, zoom control 130 performs        electronic zoom on the acquired image frame to meet the        requested zoom.

Reference is made back to FIG. 1 and referring also to FIGS. 5 a and 5b, which illustrates examples of beam splitter configurations for imageacquisition systems, according to embodiments of the present invention.In variations of the present invention, wide image acquisition device112 and tele image acquisition device 110 are coupled with a mutualfront lens 570 and a beam splitter 580, wherein one portion of the lightreaching beam splitter 580 is directed towards wide image sensor array112 and the remainder portion of the light reaching beam splitter 580 isdirected towards tele image sensor array 110. In FIG. 5 a, the beamsplitter configuration includes a wide angle lens 572, to provide imagesensor 510 a wider FOV with respect to image sensor 512. In FIG. 5 b,the beam splitter configuration includes wide angle lens 572, to provideimage sensor 510 a wide FOV, and a narrow angle lens 574, to provideimage sensor 512 a narrow FOV, relative to the FOV of image sensor 512.

Reference is now made to FIG. 3, which is a block diagram illustrationof zoom control sub-system 300 for an image acquisition system,according to some embodiments of the present invention. Zoom controlsub-system 300 includes an image sensor 310 having a lens module 320with a fixed focal length lens or a zoom lens, a zoom control module 330and a digital-zoom module 340. An object 20 is captured by image sensor310 through lens module 320. Zoom control unit 330 calculates the mostoptimal values for image sensor 310, binning/skip factors and continuousdigital-zoom values that are provided to digital-zoom unit 340. Settingthe binning/skip factor and windowing of image sensor 310 allows to keepa suitable frame refresh rate, while digital-zoom unit 340 providescontinuous zoom.

A binning function, which function is optionally provided by the sensorarray provider, is a zoom out function that merges 2×2, or 4×4, or 8×8pixels pixel array, or any other square array of pixels, into a singlepixel, whereby reducing the image frame dimensions. The binning functionmay be refined by using algorithms such as “bi-linear” interpolation,“bi-cubic” interpolation and other commonly used digital zoomalgorithms. A skip function, which function is optionally provided bythe sensor array provider, is a zoom out function that allows skippingpixels while reading frame out, whereby reducing the image framedimensions and decrease the image acquisition time.

In variations of the present invention, zoom control sub-system 100 of aimage acquisition system includes the binning/skip function capabilitiesas in zoom control sub-system 300.

Reference is also made to FIG. 4, which is a schematic flow diagramchart that outlines the successive steps of an example continuous zoomprocess 400, according to embodiments of the present invention,performed on image acquisition system, having a zoom control sub-systemsuch as zoom control sub-system 100. Process 400 includes the flowingsteps:

Step 410: providing a wide image acquisition device and a tele imageacquisition device.

-   -   Multiple optical image acquisition devices can be used, but for        description clarity, with no limitation, the method will be        described in terms of two image acquisition devices: wide image        acquisition device and a tele image acquisition device.    -   Both image acquisition devices (110 and 112) include an image        sensor array coupled with a lens (120 and 122, respectively),        providing a fixed FOV (tele FOV 140 and wide FOV 142,        respectively). Preferably, wide FOV 142 is substantially wider        than narrow FOV 140.    -   The image acquisition devices are adjacently disposed, such that        at least a portion of the environment, viewed from within narrow        FOV 140 of the tele image acquisition device 110, overlaps the        environment viewed from within the wide FOV 142 of wide image        acquisition device 112.        Step 420: determining alignment offsets.    -   Before using zoom control sub-system 100, an electronically        calibrating is performed to determine the alignment offsets        between wide image sensor array 110 and tele image sensor array        112. Typically, since the spatial offsets between wide image        sensor array 110 and tele image sensor array 112 are fixed, the        electronic calibration step is performed one time, after the        manufacturing of the image acquisition system and before the        first use. The electronic calibration yields an X-coordinate        offset, a Y-coordinate offset and optionally, a Z-coordinate        rotational offset of the correlation between wide image sensor        array 110 and tele image sensor array 112. Preferably, all three        aforementioned coordinate offset values are computed in        sub-pixel accuracy. It should be noted that for image        acquisition systems with more than two image sensors, the        electronic calibration step is performed on each pair of        adjacently disposed image sensor arrays.        Step 430: zoom selection.    -   A user of the image acquisition selects the required zoom.        Step 435: bin/skip function selection.    -   The zoom control 130 selects the bin/skip function, typically        provided by the image sensor provider, bringing the combination        of the optical zoom and the binning/skip magnification        selection, to a zoom value most proximal to the requested zoom.        Step 440: selecting an image acquisition device.    -   The zoom control 130 selects an image acquisition device,        bringing the combination of the optical zoom and the        binning/skip magnification selection, to a zoom value most        proximal to the requested zoom.        Step 450: acquiring an image frame.    -   An image frame is acquired by the selected image acquisition        device.        Step 460: performing electronic zoom on the acquired image frame        to meet the requested zoom.    -   The zoom control 130 computes the zoom factor between the fixed        zoom of the selected image acquisition device, combined with the        selected by bin/skip factor, and the requested zoom. Based on        the computed factor, zoom control 130 performs electronic zoom        on the acquired image frame to meet the requested zoom.

Reference is now made to FIG. 6, which is a block diagram illustrationof a camera system 600, according to embodiments of the presentinvention, including a color image sensor 612 having wide FOV 642 and amonochrome image sensor 610 having narrow FOV 640. The angle of view ofwide FOV 142 is typically wider than the angle of view of narrow FOV140. In some variations of the present invention, the angle of view ofwide FOV 142 is substantially equal to the angle of view of narrow FOV140.

A principal intention of the present invention includes providing acamera system 600 and a method of use thereof, wherein the output imageframe 650 has the resolution of image sensor 610, having narrow FOV 640,and the color of image sensor 612, having wide FOV 642.

Reference is now made to FIG. 7, which is a block diagram illustrationof another zoom control sub-system 700 for a color image acquisitionsystem, according to variations of the present invention. A coloredimage frame 632 is acquired by wide image sensor array 612, and amonochrome image frame 630 is acquired by narrow image sensor array 610.When image sensor selector 750 closes contact 752, monochrome imagesensor 610 is bypassed and only color image sensor 612 having is inoperation.

When image sensor selector 750 closes contact 754, both monochrome imagesensor 610 and color image sensor 612 are in operation, whereas imageframes are acquired by monochrome image sensor 610 and color of imagesensor 612, synchronously. Fusion module 660 extracts the colorinformation from color image frame 632 and fuses the extracted colorinformation with monochrome image frame 630 to form a high resolution,colored image frame 650. The fusion includes computing color values forthe high resolution pixels of monochrome image frame 630 from therespective low resolution color image frame 632. Preferably, thecomputation and alignment of the color values is performed in sub pixelaccuracy.

In some variations of the present invention, the output colored imageframe 650 is provided with RGB information. In other variations of thepresent invention, fusion module 760 transmits the Y information,obtained from monochrome image sensor 610 covered with color (Cr, Cb)information obtained from color image sensor 612. The color informationobtained from color image sensor 612 via a color space. Then, fusionmodule 760 merges the Y information, obtained from monochrome imagesensor 610, and the color (Cr, Cb) information. Then, color spaceconversion module 770 converts the image back to an RGB color space,creating colored output image frame 650. Optionally, the (Y, Cr, Cb)image information is transmitted in separate channels to an imagereceiving unit, bypassing color space conversion module 770.

The invention being thus described in terms of embodiments and examples,it will be obvious that the same may be varied in many ways. Suchvariations are not to be regarded as a departure from the spirit andscope of the invention, and all such modifications as would be obviousto one skilled in the art are intended to be included within the scopeof the claims.

1. In a computerized image acquisition system, having multiple opticalimage acquisition devices each with a fixed field of view (FOV), amethod for continuous electronic zoom comprising the steps of: a)providing a first image acquisition device including: i) a first imagesensor array coupled with a first lens having a first FOV; and ii) afirst electronic zoom; b) providing a second image acquisition deviceincluding: i) a second image sensor array coupled with a second lenshaving a second FOV; and ii) a second electronic zoom; wherein at leasta portion of the environment, viewed from within said second FOV of saidsecond image acquisition device, overlaps the environment viewed fromwithin said first FOV of said first image acquisition device; c)electronically calibrating the alignment between said first image sensorarray and said second image sensor array, whereby determining anX-coordinate offset and a Y-coordinate offset of the correlation betweensaid first image sensor array and said second image sensor array; d)providing a user of the image acquisition device with a zoom selectingcontrol, thereby obtaining a requested zoom; e) selecting one of saidimage acquisition devices based on said requested zoom; f) acquiring animage frame with said selected image acquisition device, therebyobtaining an acquired image frame; and g) performing digitally zoom onsaid acquired image frame, thereby obtaining an acquired image framewith said requested zoom, wherein said calibrating of said alignmentbetween said first image sensor array and said second image sensorarray, facilitates continuous electronic zoom with uninterruptedimaging, when switching back and forth between said first image sensorarray and said second image sensor array.
 2. The method as in claim 1,wherein the computerized image acquisition system is configured toprovide zooming functions selected from the group consisting of a binfunction and a skip function; wherein said selecting of said imageacquisition device includes selecting the parameters of said bin and/orskip functions; and wherein said method further includes the step ofapplying said selected bin/skip functions, with said selectedparameters, to said acquired image frame, before said performing of saiddigital zoom step.
 3. The method as in claim 1, wherein said imagesensor arrays are focused to the infinite.
 4. The method as in claim 1,wherein a lens, selected from the group consisting of said first lensand said second lens, is a focus adjustable lens.
 5. The method as inclaim 1, wherein a lens, selected from the group consisting of saidfirst lens and said second lens, is a focus adjustable lens.
 6. Themethod as in claim 1 wherein said second lens is a zoom lens.
 7. Themethod as in claim 1, where said electronic calibration of saidalignment between said first image sensor array and said second imagesensor array, further determines a Z-coordinate rotational offset of thecorrelation between said first image sensor array and said second imagesensor array.
 8. The method as in claim 1, wherein said electroniccalibration is performed with sub-pixel accuracy.
 9. The method as inclaim 1, wherein said electronic calibration step is performed on eachpair of adjacently disposed image sensor arrays.
 10. The method as inclaim 1, wherein said first image acquisition device and said secondimage acquisition device are coupled with a mutual front lens and a beamsplitter, wherein one portion of the light reaching said beam splitteris directed towards said first image sensor array and the remainderportion of the light reaching said beam splitter is directed towardssaid second image sensor array.
 11. The method as in claim 1, whereinthe angle of view of said first FOV is wider than the angle of view ofsaid second FOV.
 12. The method as in claim 1, wherein said first imagesensor array is a color sensor and said second image sensor array is amonochrome sensor, wherein a colored image frame is acquired by saidfirst image sensor array; wherein a monochrome image frame is acquiredby said second image sensor array; and wherein said colored image frameand said monochrome image frame are fused to form a high resolutioncolored image frame.
 13. The method as in claim 12, wherein said fusionof said colored image frame and said monochrome image frame includes thestep of computing color values for the pixels of said monochrome imageframe from the respective pixels of said colored image frame.
 14. Themethod as in claim 12, wherein the angle of view of said first FOV iswider than the angle of view of said second FOV.
 15. The method as inclaim 12, wherein the angle of view of said first FOV is substantiallyequal to the angle of view of said second FOV.
 16. The method as inclaim 13, wherein said computing of color values is performed in subpixel accuracy.
 17. The method as in claim 2, wherein said image sensorarrays are focused to the infinite.
 18. The method as in claim 2,wherein said second lens is a zoom lens.
 19. The method as in claim 7,wherein said electronic calibration is performed with sub-pixelaccuracy.